System and methods for safety, security, and well-being of individuals

ABSTRACT

A system includes video cameras arranged to monitor a vulnerable person, and a processor system that receives video frames from the video cameras, the processor system comprising a processor and a non-transitory, computer-readable storage medium having machine instructions executed by the processor. The processor detects and identifies objects in a current received video frame, classifies an identified object as the person by applying a facial recognition algorithm that identifies the person, determines a posture of the person by identifying joints, limbs, and body parts, and their respective orientations to each other and to a plane, and immediately discards the current video frame. The processor then determines a change in motion, of the person, between the current received video frame and one or more prior received video frames, and, based on the determined posture and the change in motion, determines that the person has experienced a defined event.

RELATED APPLICATIONS

This application claims priority to provisional patent application63/066,296 filed Aug. 16, 2020, entitled System and Method for Safety,Security and Well-Being of Individuals, the disclosure of which ishereby incorporated by reference.

BACKGROUND

Certain individuals (persons) may require assistance and/or monitoringto ensure their security, safety, and/or well-being. For example,vulnerable individuals, such as elderly individuals with physical ormental impairments, may require some form of assisted living. Other,non-elderly, vulnerable individuals also may require some form ofassisted living. Such assisted living may take place in a long-termassisted living facility. Alternately, such assisted living may takeplace in the vulnerable individuals' homes.

Whether in an assisted living facility or a home, vulnerableindividuals, such as those with physical impairments, may be at risk ofinjury from falls or other injuries while alone. Individuals withcognitive impairments, when alone, may be at risk from wandering, falls,and use of dangerous objects, or may suffer from panic or confusion.Some vulnerable individuals may be bed-ridden or otherwise have limitedmobility, and such individuals may be subject to pressure ulcers,commonly referred to as bed sores.

Other vulnerable individuals may require short-term assistance, such asmay be provided at a health care facility or hospital by a nurse, amedical technician, or another medical professional or caregiver.

In addition to population-based problems, there are also universalproblems in the care of vulnerable individuals, experienced by thevulnerable individual and the caregiver. Universal challenges to thecaregiver include time and attention requirements, and monetary costs.The constant, uninterrupted monitoring of vulnerable individuals by ahuman caregiver is impractical, expensive, and sometimes infeasible. Inthe setting of a home, a family member or hired caregiver may not alwaysbe available to physically watch the vulnerable individual. It is bothlogically impractical and infeasible to always have eyes on thevulnerable individual because the caregiver will have to carry out otherduties. For instance, the caregiver may have to prepare medications,meals, or run errands. Additionally, a hired caregiver can be too costlyfor many families to afford. In the setting of a formal care facility,such as independent living facilities, assisted living facilities,nursing homes, or hospitals, the number of vulnerable individuals makesit practically impossible for them to be monitored one on one by staffat all times.

Universal challenges to the vulnerable individual are privacy andautonomy. Caregivers and healthcare professionals often worry about therisk of allowing independence of a vulnerable individual. Risk of a falland subsequent injury in a caregiving environment or restroom maynecessitate the presence of a caregiver, and is but one example thatresults in the loss of a vulnerable individual's independence andprivacy.

Current technologies aimed at monitoring vulnerable individuals includewearable pendants, ambient sensors (pressure sensors, motion sensors,magnetic sensors, bed alarms), and camera-based sensors. Thesetechnologies all share certain drawbacks which include: they are limitedto a specific number of functions, do not provide any higher-levelinformation, and most are reactive rather than predictive. Due to thelimited functionality of these technologies, this may necessitate theuse of multiple devices which becomes invasive, costly, and cumbersome,and relies on the compliance of individuals to use them (i.e., wearablependants).

These existing technologies also has specific drawbacks. Wearablesystems use sensors such as accelerometers and gyroscopes in the form ofa pendant, belt, or watch to detect a fall. While they may be accurateand may send real time alerts, these systems depend on the individual towear the sensor. Many individuals forget to wear the sensor, or choosenot to wear it because the sensor is uncomfortable or cumbersome.Additionally, some sensors require individuals to manually activate analarm after a fall. An individual who has fallen may not have thecognitive or physical capacity to activate an alarm.

Ambient sensors include sensors such as motion sensors, pressuresensors, and magnetic sensors. These sensors can detect a specific formof activity, such as but not limited to, opening a door (magneticsensor), standing up from bed (pressure sensor), or entering a room(motion sensor). While ambient sensors may provide some utility, theyare expensive, invasive, and cumbersome to install multiple sensors tocover the desired area for monitoring.

Current camera-based monitoring systems may be used to monitorvulnerable individuals. One such system utilizes multiple cameras tocover areas of interest and to provide a video feed to a group ofindividuals dedicated to remotely watching the camera feed andmonitoring for an adverse event, such as a fall. For this reason,current camera-based systems cannot provide privacy, and the constantmonitoring fatigues the human observers and increase the cost ofmonitoring. Other camera-based systems that utilize artificialintelligence-based detection algorithms instead of human observers alsodo not address the problem of privacy because they still allow humanobservers to view the video footage.

To set up a home or healthcare facility with current technologies,several modalities may be required to adequately protect the vulnerableindividual. For example, to ensure a vulnerable individual does not getout of bed, a bed alarm or pressure sensor must be used. To monitor thevulnerable individual for falls, a wearable device must be used. Toensure the individual does not leave a particular room or the home,multiple motion sensors or magnetic sensors for doors/windows must beused. To gain insight as to what an individual is doing from a remotelocation, the caregiver must watch a real-time or near-real-time videofeed from a camera. Automated camera-based systems exist that allow forbasic event detection and alerts such as motion detection and dooropening, the current camera-based systems do not provide adequatefunctionality for monitoring vulnerable individuals. That is, currentautomated camera-based systems are not designed or adapted forcaregiving or for healthcare facilities. Therefore, existing automatedcamera-based systems, by themselves, are not able to provide adequatefunctionalities to monitor vulnerable individuals.

SUMMARY

An example automated video monitoring system includes a plurality ofvideo cameras arranged to monitor a vulnerable person, and a processorsystem configured to receive video frames from a video stream providedby the plurality of video cameras. The processor system may include oneprocessor or multiple processors. The processor system includes anon-transitory, computer-readable storage medium having encoded thereonmachine instructions executed by the processors. A processor executesthe machine instructions to recognize and identify objects in a currentreceived video frame, and to classify an identified object as thevulnerable person. To classify an object, the processor may apply afacial recognition algorithm that identifies, by name, the vulnerableperson, and determines a posture of the named vulnerable person byidentifying joints, limbs, and body parts, and their respectiveorientations to each other and to a horizontal plane, determines achange in motion of the named vulnerable person between the currentreceived video frame and one or more prior received video frames, andbased on the determined posture and the change in motion, determinesthat the named vulnerable person has experienced a defined event.Immediately following, the processor discards the current received videoframe.

An example system for monitoring the safety, security, and well-being ofa person includes a non-transitory, computer-readable storage mediumhaving encoded thereon, machine instructions executable by a processorto monitor the safety, security, and well-being of the person, where theprocessor receives raw image data of video frames captured by one ormore of a plurality of video cameras; detects and identifies imageobjects in a current received video frame; classifies an identifiedimage object as a vulnerable person, comprising the processor applying afacial recognition algorithm to the identified image object; determinesa posture of the vulnerable person, comprising identifying a pluralityof body parts of the vulnerable person, and the respective orientationsof the body parts to each other; automatically and immediately followingdetecting and classifying the image object and determining the posture,deletes the current video frame; and based on the determined posture,determines that the vulnerable person may experience a first definedevent. In an aspect, to determine a posture of the vulnerable person,the processor compares the relative orientations of the body parts toone or more of a vertical plane and a horizontal plane. In an aspect, todetermine a posture of the vulnerable person, the processor generates asegmented room comprising one or more room segments, a room segmentcorresponding to a surface of the segmented room; and determines arelationship between the vulnerable person and one or more of the roomsegments. In an aspect, a segment comprises a floor segment of thesegmented room, and wherein the image object comprising the vulnerableperson contacts at least a portion of the floor segment. In an aspect,the processor determines an amount of change in posture of thevulnerable person between the current received video frame and one ormore prior received video frames; and based on the determined change inthe amount of posture of the vulnerable individual, determines thevulnerable person has experienced a second defined event, wherein thesecond defined event is a fall. In an aspect, the processor determinesan amount of change in posture of the vulnerable person between thecurrent received video frame and one or more prior received videoframes; and based on the determined change in the amount of posture ofthe vulnerable individual, determines the vulnerable person mayexperience a third defined event, wherein the third defined event is apressure ulcer. In an aspect, to classify the identified object as thevulnerable person, the processor generates a bounding box encompassingthe identified image object; and applies the facial recognitionalgorithm only to the identified image object encompassed in thebounding box.

A method for monitoring safety, security, and well-being of a personincludes a processor: receiving raw image data of video frames capturedby one or more of a plurality of video cameras; detecting andidentifying image objects in a current received video frame; classifyingan identified image object as a vulnerable person, comprising theprocessor applying a facial recognition algorithm to the identifiedimage object; determining a posture of the vulnerable person, comprisingidentifying a plurality of body parts of the vulnerable person, andrespective orientations of the body parts; automatically and immediatelyfollowing detecting and classifying the image object and determining theposture, deletes the current video frame; and based on the determinedposture, determining that the vulnerable person may experience a firstdefined event.

DESCRIPTION OF THE DRAWINGS

The detailed description refers to the following figures in which likenumerals refer to like objects, and in which:

FIG. 1 illustrates a video frame capture of an example environment inwhich an example video monitoring system may be implemented;

FIG. 2 is a block diagram illustrating the example video monitoringsystem of FIG. 1;

FIG. 3 is a block diagram of software components of the example videomonitoring system;

FIG. 4 is a block diagram of another example video monitoring system;

FIGS. 5-8 represent video frames captured by the video monitoring systemduring an event occurring in the example environment of FIG. 1;

FIGS. 9-12 represent analyzed video frames captured by the videomonitoring system during an event occurring in the example environmentof FIG. 1;

FIGS. 13-16 represent contextualized and analyzed video frames capturedby the video monitoring system during an event occurring in the exampleenvironment of FIG. 1;

FIGS. 17A and 17B illustrate aspects of the analysis andcontextualization operations executed by the video monitoring system ofFIG. 2;

FIGS. 18-20 illustrate example information displays generated by thevideo monitoring system of FIG. 2;

FIGS. 21-22 are flowcharts illustrating example operations of the videomonitoring system of FIG. 2;

FIG. 23 illustrates and example image object; and

FIGS. 24-26 are flowcharts illustrating additional example operations ofthe video monitoring system of FIG. 2.

DETAILED DESCRIPTION

Disclosed herein is a system for predicting an occurrence of eventsharmful to a monitored, vulnerable person includes a plurality of videocameras arranged to monitor the vulnerable person; and a processorsystem configured to receive video frames from a video stream providedby the plurality of video cameras, the processor system comprising oneor more processors and a non-transitory, computer-readable storagemedium having encoded thereon machine instructions executed by the oneor more processors. The processor detects and identifies image objectsin a current received video frame, classifies an identified object asthe vulnerable person, which comprises applying a bounding box to theidentified object; and applying a facial recognition algorithm only tothe identified object encompassed by the bounding box, the facialrecognition algorithm comparing facial data to a database of facialimages including the vulnerable person. The processor further determinesa posture of the vulnerable person, comprising identifying joints,limbs, and body parts, and their respective orientations to each otherand to one or more of a horizontal plane and a vertical plane,immediately following the detecting, classifying, and determining,deleting the current frame, and based on the determined posturedetermines that the vulnerable person may experience a defined event.

Further, disclosed herein is a method for monitoring safety, security,and well-being of a person includes a processor: receiving raw imagedata of video frames captured by one or more of a plurality of videocameras; detecting and identifying image objects in a current receivedvideo frame; classifying an identified image object as a vulnerableperson, comprising the processor applying a facial recognition algorithmto the identified image object; determining a posture of the vulnerableperson, comprising identifying a plurality of body parts of thevulnerable person, and respective orientations of the body parts;automatically and immediately following detecting and classifying theimage object and determining the posture, deletes the current videoframe; and based on the determined posture, determining that thevulnerable person may experience a first defined event. In an aspect,the processor determines a posture of the vulnerable person by comparingthe respective orientations of the body parts relative to one or more ofa vertical plane and a horizontal plane. In another aspect, theprocessor determines a posture of the vulnerable person, by generating asegmented room comprising one or more room segments, a room segmentcorresponding to a surface of the segmented room; and determining arelationship between the vulnerable person and one or more of the roomsegments. In an aspect, a segment comprises a floor segment of thesegmented room, and the image object that is classified as thevulnerable person, contacts at least a portion of the floor segment. Inan aspect, the processor determines determining an amount of change inposture of the vulnerable person between the current received videoframe and one or more prior received video frames; and based on thedetermined amount of change in posture of the vulnerable person,determining the vulnerable person has experienced a second definedevent, wherein the second defined event is a fall. In another aspect,the processor determines an amount of change in posture of thevulnerable person between the current received video frame and one ormore prior received video frames; and based on the determined amount ofchange in posture of the vulnerable person, determining the vulnerableperson may experience a third defined event, wherein the third definedevent is a pressure ulcer. In an aspect, the processor classifies theimage object as the vulnerable person by generating a bounding boxencompassing the identified image object; and applying the facialrecognition algorithm only to the identified image object encompassed inthe bounding box. In an aspect, the processor generates a confidencelevel for an accuracy of the classifying; determines the generatedconfidence level is less than a desired confidence level; and deletesthe image object. In an aspect, the first defined event is a fall, andthe processor generates a potential fall alert; and provides thepotential fall alert to a caretaker attending to the vulnerable person.

Still further, disclosed herein is a video monitoring system (andcorresponding methods), that allows automated detection of eventsrelated to caregiving and health care, such as, but not limited to,falls, pressure ulcers, wandering, postural hypotension, use ofdangerous objects (stove, fireplace, knives, etc.), seizures, andchoking, while maintaining the privacy of vulnerable individuals (thatis, vulnerable persons) in intimate situations such as, but not limitedto, a bathroom, a hospital room. The video monitoring system, inaddition to automated detection of events, also implements a predictivefunction that may warn caregivers that a possible adverse event mayoccur to a vulnerable person.

The description of this video monitoring system, with its automatedevent detection and automated event prediction, begins with reference toa specific scenario involving long-term health care of a vulnerableperson by a dedicated caregiver. However, the video monitoring system isnot limited to this specific scenario, as one skilled in the art willreadily recognize. In addition, implementation of the video monitoringsystem, and its corresponding use, is described with reference to otherscenarios.

In describing the video monitoring system in the context of a healthcare scenario, the detailed description refers to the following termsand their associated meanings:

Vulnerable person, as used herein, is a human of any age who needs or isprovided with some form of health care, or other type of care, for aprescribed period, which may be short-term health care such as might beexpected at a hospital (although some long-term health care scenariosmay exist at a hospital) or long-term health care which may be providedin the vulnerable person's home, at an assisted living center, nursinghome, or other long-term health care facility. During the prescribedperiod (e.g., weeks, months, years), the health care may be provided 24hours per day, or only for a portion of a day.

Caregiver, as used herein in some scenarios, is a medical professionalsuch as a nurse, doctor, or medical technician, or other person assignedto tend to, assist, and care for a vulnerable person. A caregiver may beassigned to a single vulnerable person or to multiple vulnerablepersons. In other scenarios, the role and functions of a caregiver maybe filled by a guard or similar observer.

Object, as used herein may be a human (e.g., a vulnerable person or acaregiver), may be an animal, or may be inanimate. An object may referto a thing (e.g., another human, an animal, or an inanimate object) thata vulnerable person may encounter during a period of the provided healthcare. Inanimate objects may be fixed or capable of motion. An inanimateobject may have a fixed structure, orientation, or shape or a changeablestructure, orientation or shape. The herein disclosed video monitoringsystem includes features and components that can distinguish betweenindividuals (humans), animals, and inanimate objects.

Observer, as used herein, when employed, is distinct from a caregiver.An observer is a human who may monitor or view information related to avulnerable person. When employed, an observer may cooperate with acaregiver to ensure the safety, security, and well-being of a vulnerableperson. An observer may, but need not, be collocated with the caregiver.In scenarios in which an observer is not employed, some or all functionsof the observer may be implemented by the video monitoring system in anautomatic and autonomous fashion, and/or may be executed by thecaregiver.

The herein disclosed automated video monitoring system (andcorresponding method) provides individualized risk management, metrictracking, and event prediction and detection while at the same time,protecting privacy of vulnerable persons in health care scenarios aswell as in other scenarios. In an example, the system captures imagedata (i.e., images of objects and data defining and characterizing theimages, and thus the objects) in video frames from a live video feed.The system then employs facial recognition (for human objects) andobject detection software to generate a higher-level representation ofthe images, and employs algorithms to identify and analyze specificfeatures and interactions of objects and persons of interest (e.g.,vulnerable persons, caregivers, etc.) within the captured video frames.The system then compares the analyzed features and interactions withpredetermined parameters of events of interest, evaluates the occurrenceof specific events based on comparison algorithms, and generatesappropriate notifications to third-parties, such as caregivers. In anaspect, the system may execute predictive algorithms to forecast apotential event of interest, and to provide appropriate notifications tothird-parties. In an aspect, the system may store specificoutcomes-measures. Finally, the system deletes the original image datain the capture video frames as soon as data required for the abovefunctions is extracted from the video frames.

The video monitoring system may be employed for monitoring of vulnerableindividuals in private or intimate settings (i.e., in the home,bathrooms, changing rooms, etc.) because, through its operations,information-rich data may be extracted while protecting the privacy ofthe vulnerable individual. These system features allow for: (1)real-time alerts to third-parties if an emergency/high risk event isdetected or predicted (i.e., falls, seizures), and (2) evaluation oflongitudinal, personalized metric tracking for outcome measures ofinterest (e.g., steps taken/distance moved for elderly persons,positional changes in bedridden persons, developmental milestones inchildren).

The video monitoring system may be implemented in hospitals, privatehomes, and assisted-living facilities. The system and correspondingmethod represent significant technological advances over current systemsand methods. The system and corresponding method providetechnologically-advanced remote event monitoring by third-parties ofvulnerable persons that allows rapid response to adverse events withoutconstant/continuous human observation. The advanced remote eventmonitoring ensures the safety, security, and well-being of vulnerablepersons while simultaneously protecting their privacy and autonomy. Thesystem's longitudinal metric tracking may be applied to future researchor safety initiatives in order to improve health interventions andoutcomes for the monitored vulnerable persons.

FIG. 1 illustrates a video frame capture of an example environment inwhich an example video monitoring system may be implemented. In FIG. 1,environment 100 (i.e., room 100) is a private resident room of ahospital in which vulnerable person 110 (i.e., a patient) isrecuperating. The room 100 can be seen to have a door opening to ahallway of the hospital and a door opening to a private restroom.Objects in the room 100 include bed 121, clock 122, chairs 123 and 124,and floor 125. The image represented in FIG. 1 is a single video frame(i.e., a raw image) captured by a camera (not shown in FIG. 1), which isa component of the video monitoring system (also not shown in FIG. 1).Overlaying the video frame may be a camera identification (in theexample of FIG. 1, the camera is identified as camera 1) as well as aframe sequence number (not shown). In the disclosure that follows, ourpatient, or vulnerable person 110, James Williams, will experience oneor more “events” while in the example environment 100, and thedisclosure describes how those “events” are detected, analyzed,contextualized, and reported through operation of the video monitoringsystem.

FIG. 2 is a block diagram of hardware components of the example videomonitoring system. In FIG. 2, example video monitoring system 200includes one or more video cameras 210, one or more processor systems220, one or more data transmission systems 240, and one or more displaysystems 250. A camera 210 may be capable of capturing a wide field ofview. A camera 210 may be configured to operate in multiple illuminationstates including very low light states. A camera 210 may include anormal wired power supply 211 as well as a back-up battery power supply211 b. The wired power supply may be hard-wired. A camera 210 may beprogrammed for continuous recording or may be programmed to record basedon the occurrence of some detected activity within its field of view.For this later capability, a camera 210 may include a motion detector212 and/or a microphone 213. A camera 210 may include a datatransmission mechanism 214 that provides for communication with theprocessor system 220. The data transmission mechanism 214 may be used totransmit video frames to the processor system 220. The data transmissionmechanism 214 may be configured to transmit data other than video framesto the processor system 220. In an aspect, video frames to betransmitted to the processor system 220 may be transmitted in real time;that is as soon as acquired and without buffering in the video camera210. However, the video camera 210 may incorporate a buffer to holdvideo frames in a situation when data traffic slows reception at theprocessor system 220. In an example, the video monitoring system 200 mayemploy multiple processor systems 220 to accommodate expected dataflows. A processor system 220 may include a processor 222, memory 224,data store 226, and data bus 228. The data store 226 may includenon-transitory, computer-readable storage medium 227 having encodedthereon, a machine instruction set for executing the functions of thevideo monitoring system 200. The data store 226 also may store data 230extracted by operation of the processor system 220 from the capturedvideo frames; however, such stored data does not include any videoimages from the video frames. The data store 226 may store digitalimages of vulnerable persons 110 and of caregivers to be used for facialrecognition. These digital images of vulnerable persons may be acquiredfrom the vulnerable person's medical file. An example machineinstruction set is disclosed in more detail herein, including withrespect to the description of FIG. 3. The data transmission system 240may be configured as a wired local area network, for example, thatcouples cameras 210 to processor systems 220. Alternately, the datatransmission system 240 may be implemented as a wireless network. In anexample, some components of the video monitoring system 200 may belocated remotely (e.g., in the cloud). The display system 250 includeshardware components that may provide visual and/or audible alerts tocaregivers and other authorized observers and medical professionals.However, in an example, data presented on the display system 250 areconfined only to analyzed and contextualized data, and alerts, anddisplay system 250 does not display video frames. An example hardwarecomponent of the display system 250 may be a wearable display devicesuch as a smartwatch having a specific application installed. Otherhardware components include standard monitors available, for example, ata hospital's nursing station or similar location.

FIG. 3 is a block diagram of software components of the example videomonitoring system. In FIG. 3, example software system components 300includes video frame processor 305, object recognition processor 310,facial recognition processor 315, posture recognition processor 320,data analyzer 325, context analyzer 330, display driver 335, alertmodule 340, and remote data display application 345. The components 300(with the exception of application 345) may be loaded into memory 224from data store 226 to be executed by processor 222 (see FIG. 2). Acomponent 300 of the software system is described briefly as follows,and in more detail elsewhere in this disclosure. Video frame processor305 receives a video frame captured by video camera 210, indexes thevideo frames, records the video source and time received, and executesany required pre-processing operations and image enhancement operationsincluding decryption (if received encrypted), pixel interpolation (ifreceived compressed), gamma correction, and other routine digital imageprocessing operations. The video frame processor 305 may receive videoframes at a rate of approximately 5-10 frames per second, and mayprocess (i.e., extract all required data) in a frame in a time of about0.05-0.1 seconds. Multiple processors 222 may be used when multiplecameras are employed as well as to increase the number of framesprocessed per second from a single camera.

Object recognition processor 310 operates to recognize objects in theenvironment (e.g., the environment 100 of FIG. 1) using trained objectrecognition algorithm 312 and object motion or tracking algorithm 314.The algorithm 312 may be trained using, for example, a neural network,to recognize humans, animals, and inanimate objects. In an aspect, thealgorithm 312 may be trained to recognize specific vulnerable personsand specific caregivers. Object recognition algorithm 312 is executed tofirst, detect discrete objects in the environment 100, second, identifyall discrete detected objects in the environment 100, and third todetermine locations of the identified objects in the environment 100through use of a bounding box in the image. An aspect of objectrecognition algorithm 312 is executed to identify as humans (i.e.,individuals), certain objects initially detected as objects in theenvironment 100. In this aspect, the algorithm 312 may comparecharacteristics of a detected object to a database of human forms, thedatabase of human forms including forms for males, females, adults,children, and forms of such humans in various postures, includingstanding and siting, for example. To improve the accuracy of objectrecognition, the object recognition processor may employ boundary boxes(see, e.g., FIGS. 17A and 17B) in which the object is, or is expected tobe located. For example, the algorithm 312 may be applied to a boundingbox to multiple objects in the environment 100, and the algorithm 312takes into account the relationship of the bounding boxes of themultiple objects relative to a specific bounding box when attempting toidentify an object in the specific bounding box. Then, when attemptingto identify an object in the specific bounding box, the algorithm 312uses image data only for the image created by the specific bounding box,using that image data to search for the object within the specificbounding box. Use of such boundary boxes for static object recognitionare known in the art. The object recognition algorithm 312 may betrained to recognize a human object based only on detection of theperson's face, since in some scenarios, the entire human form, or asignificant portion of the human form, may not be directly visible in avideo frame.

Object tracking algorithm 314 executes to track or follow an identifiedobject in the environment 100. The object tracking algorithm 314combines information from a history of known object locations withinenvironment 100 and computes how an object moves between video frames.Because some objects may change shape or move, boundary boxes used withalgorithm 312 may be adapted to move and/or change shape as objects moveor change shape. Object tracking provides information about anidentified object in the video frame, and how an object moves. A videoframe may be time-stamped (with, for example, the time registered forthe video frame), annotated with camera identification, and saved forsubsequent use by components of the processor system 220. In an aspect,the object tracking algorithm 314 receives as an input, results of aroom segmentation operation. For example, considering FIG. 1, room 100can be seen to include a number of surfaces with which objects,including Mr. Williams 110 may interact. One such surface is floor 125,and Mr. Williams 110 may be standing on the floor 125, with an uprightor a bent over posture, may be sitting on the floor 125, or may be lyingon the floor 125. Other postures are possible. One purpose of segmentinga room is to allow the algorithm 314, and other aspects of theprocessing system 220, to better track or follow a both a generallystationary object such as bed 121, as well as moveable objects, such asa person, from frame to frame. Continuing with this example, bed 121should generally align with, or interact with floor 125. For example,were the bed 121 to be viewed by the processor system 220 as attached toa wall, an obvious error condition would—one possible cause would beinadvertent or unintended movement of camera 1 in FIG. 1. Similarly, infollowing Mr. Williams 110 frame to frame, the algorithm 314 may receivean input as to which, if any, room segments (e.g., floor 125) Mr.Williams has contacted.

Facial recognition processor 315 operates to recognize vulnerablepersons such as vulnerable person 110 of FIG. 1 using trained facialrecognition algorithm 317 (the algorithm 317 will be seen later toidentify our vulnerable person 110 as James Williams). The facialrecognition processor 315 also may recognize other individuals such ascaregiver 140 (see FIG. 17A). To identify an individual, the algorithm317 is executed using a pre-existing database of facial identities. Inthe environment 100, Mr. Williams' face may be registered in thepre-existing database of facial identities. Additionally, caregiverssuch as caregiver 140 of FIG. 17A, may have their faces registered inthe pre-existing database of facial identities. If the person's facematches one of the facial identities in the database, the person isidentified. If the face of the person does not match one of theidentities in the facial identities database, or if the person is notfacing the camera 210, the person may be designated as an unknownperson. The person's identity status, known or unknown, and when known,the actual name or other identification of the known person isassociated with the corresponding object, and stored for subsequent useby the processor system 220.

Posture recognition processor 320 operates to recognize the posture(e.g., sitting, standing, bent over, lying) of the vulnerable person, orother person, using trained posture recognition algorithm 322. Thealgorithm 322 may be trained using a neural network. The processor 222executes posture recognition algorithm 322 to identify and track thepostures of persons identified by facial recognition as well asunidentified persons in environment 100. The algorithm 322 is executedto analyze the position of joints and anatomical landmarks in relationto each other to determine the posture of the person. Since the locationof a person in the video frame has been established already, thealgorithm 322 may be executed to apply postural analysis to just theportion of the video frame (raw video image) containing the person,allowing for a reduction in overall computation load and to improve theaccuracy of the posture analysis. To effectuate this reduction incomputational load, the algorithm 322 may use or establish a boundarybox, such as those shown in FIGS. 17A and 17B, encompassing the person.Motion algorithm 324 is executed to compare the person's posture in thecurrent video frame to the person's posture in previous video frames inorder to compute how a person is moving or changing posture. Forexample, the algorithm 324, in cooperation with algorithm 312, maydetect movement of joints frame-to-frame, such as a knee joint that isbent 90 degrees in a first frame and is straight in the next frame or asubsequent frame. The rapidity of joint (and posture) change may bedetermined based on the number of frames over which joint motion (i.e.,joint flexion, posture change) is detected. Thus, an event of a personfalling may occur over fewer frames than the same person purposelymoving from a standing position to sitting or lying on the floor. Theperson's posture and posture changes may be saved for subsequent use bythe processor system 220. Furthermore, in an aspect, the algorithm 324may receive an input relating the vulnerable person 110 to a verticalplane (i.e., a plane parallel to the wall segments of room 100) and/or ahorizontal plane (i.e., a plane parallel to the floor segment (floor125) of room 100). Such inputs may improve the accuracy of the postureanalysis process.

Data analyzer 325 receives outputs from the processors 315 and 320 todetermine a current status of vulnerable person 110 using, for example,pre-defined rule set 327.

Context analyzer 330 receives the current status of vulnerable person110 as well as the output of processor 310 to determine if vulnerableperson 110 has experienced a defined event, or to predict if vulnerableperson 110 may experience a defined event.

In operation, the data analyzer 325/context analyzer 330 of theprocessor system 220 cooperate to combine the information collected (thelocations and movement of the objects, the person's identity, and theperson's posture and movement) to make decisions about the scene byapplying predetermined ruleset(s). When data present in a current videoframe are not sufficient to recognize an object, especially a humanobject, the data analyzer/context analyzer 330 may access data fromprior frames to confirm that the identity of the object. To support this“look back” analysis, the processor system 220 may store prior videoframe extracted data in data store 226. To reduce storage demands, thedata store may retain data according to the time since extraction. Forexample, data from the last one minute of video frames may be stored indetail while earlier data may be aggregated or summarized. In an aspect,the data stored in data store 226 may be transferred to a long-termstorage facility (not shown). Predetermined ruleset(s) (e.g., rule set327) may be customized for different scenarios, person, andenvironments. Predetermined rulesets may be designated to specific knownor identified persons, all persons, or unknown persons. A ruleset may beused to check a set of predetermined conditions, and if the conditionsare met, may present an alert requirement to the alert module 340.

Display driver 335 provides the graphical displays, text alerts andaudio alerts for display on devices operated or monitored by caregiver140 and other personnel charged with ensuring the safety, security, andwell-being of vulnerable person 110.

The alert module 340 generates an alert to the appropriate caregiver orobserver. The alert may provide a specific textual description of thevisual information captured by the video camera 210. The textualdescription allows for the preservation of critical information obtainedvisually, without a human observer to view captured video footage. Thealert may be sent in multiple forms, including but not limited to acall, text, a visual display, and audible signal, or a pushnotification, or combinations of these forms to a mobile device, anemail, a pager, or a local alarm.

Remote data display application 345 may operate in conjunction withsystem components 300 and may be loaded onto a portable display devicesuch as a smartwatch worn by caregiver 140.

FIG. 4 is a block diagram of another example video monitoring system. InFIG. 4, video monitoring system 350 includes combined video camera(s)352 and processing system(s) 354. The video monitoring system 350further includes a data transmission system 356 linked to the processingsystem 354. Video camera 352 is similar in most respects to video camera210 of FIG. 2. However, video camera 352 buffers (in buffer 353) videoframes acquired during video monitoring operations and provides thecaptured video frames to the processing system 354 on a rolling basis.Video processing system 354 is built into the video camera 352 such thatcaptured video frames never leave the confines of the video camera 352.Thus, all processing executed by the processor system 220 of FIG. 2 andthe software system components 300 of FIG. 3 is completed within thevideo camera 352 by processing system 354, and only analyzed andcontextualized data are provided to other components of the videomonitoring system 350. The analyzed and contextualized data generated bythe processing system 354 is sent to the data transmission system 356,where such data are prepared for and transmitted to one or more displays358 to display the analyzed and contextualized data, and any alerts, toa human observer or caregiver 140 (see FIG. 17A). The transmitted datamay be encrypted for transmission and may be sent wirelessly, or over awired network (not shown). In an aspect, the transmitted data (e.g., anevent alert, or an event forecast) may be sent to a wearable displaydevice (smartwatch with an appropriate application, such as application345 (see FIG. 3)) worn by caregiver 140. In an aspect, data displayed onthe displays 358 may include the name of the concerned person (e.g., Mr.Williams). In an aspect, the person's name may be toggled off.

FIGS. 5-8 represent raw video frames captured by the video monitoringsystem 200 during an event occurring in the example environment 100 ofFIG. 1. In FIG. 5, our vulnerable person 110, James Williams, is shownsitting up in bed 121. In FIG. 6, Mr. Williams has changed posture andmoved to the edge of bed 121 as if getting ready to stand up. In FIG. 7,Mr. Williams is standing up on floor 125, but is not completely upright,which posture might presage a fall. In FIG. 8, Mr. Williams is lying onfloor 125. However, FIG. 8 does not provide sufficient information toallow the processor system 220 to determine if Mr. Williams fell to thefloor 125. Note, however, that given normal video camera frame rates, anumber of video frames may exist between those shown in FIGS. 7 and 8,and those intervening video frames might indicate, through postureanalysis, that Mr. William had fallen to the floor 125.

FIGS. 9-12 represent aspects of analysis of raw video frames captured bythe video monitoring system 200 during an event occurring in the exampleenvironment 100 of FIG. 1. Note that in FIG. 9, the processor system 220has yet to identify the object in the bed 121 as Mr. Williams. However,the processor system 220 has at least determined that there is adistinct object in the bed 121 and that that object is a human (or aperson), and so the processor system 220 designates the person as“unknown person” 110 a. To get to this designation, the processor system220 may first detect an object in the bed 121. Such detection may resultfrom application of object recognition algorithm 312 to the video frameshown in FIG. 9. In an aspect, the algorithm 312 has been trained torecognize a bed (bed 121). The algorithm 312 then would detect someobject in close relation to bed 121. Using, for example, a backgroundsubtraction process, the algorithm 312 may subtract the pixelsconforming to the bed 121 leaving an object whose shape the algorithm312 then recognizes as that of a human (i.e., an individual). Ratherthan, or in addition to, pixel subtraction, the algorithm 312 may useedge detection techniques to identify the object on the bed 121 as adistinct object and further to identify the object as an individual. Ofcourse, in many situations, a person lying in bed 121 would be covered,thereby complicating the processing of recognizing the presence of anobject and then identifying the object as an individual. However,complicating the process does not prevent reaching the same conclusion,and the algorithm 312 may be trained to recognize a covered person. Tofurther analyze the detected but unknown person 110 a, the posturerecognition algorithm 322 may construct boundary box 111 to encompassall recognized portions of the person 110 a. This bounding box does notactually appear in the video frame, and is shown in FIG. 9 (and in otherfigures herein) for ease of description. However, the bounding box 111is referenced to the video frame; for example, the boundaries of thebounding box 111 may be stated in terms of pixel positions. Using abounding box may limit the computational load placed on the processorsystem 220. In addition to the bounding box 111, the processor system220 may generate nodes 113 corresponding to the joints of the person 110a with lines connecting the nodes. As with the bounding box 111, thenodes and lines do not exist in the video frame and are shown for easeof description. Application of the object recognition algorithm 312 andthe posture recognition algorithm 322 indicates that the person 110 a isin a semi-sitting position on the bed 121.

FIG. 10 shows our vulnerable person 110 as James Williams, his identitydetermined by application of facial recognition algorithm 317. FIG. 10also shows Mr. Williams sitting further up in bed 121. FIG. 11 shows Mr.Williams as out of bed 121, standing on floor 125, and hunched over.FIG. 12 shows Mr. Williams lying on floor 125.

FIGS. 13-16 represent contextualized and analyzed video frames capturedby the video monitoring system 200 during an event occurring in theexample environment 100 of FIG. 1. FIGS. 13 and 14 show Mr. Williams inbed 121, transitioning from a lying to a sitting posture. FIG. 15 showsMr. Williams standing by a side of bed 121. FIG. 16 shows Mr. Williamslying on floor 125. Posture and context analysis of video framesincluding and between those shown in FIGS. 15 and 16 would providesufficient information to show that Mr. Williams fell. For example,posture recognition algorithm 322 and motion algorithm 324 would showMr. Williams moving to the floor 125 with a velocity indicative offalling.

FIGS. 17A and 17B illustrate aspects of the analysis andcontextualization operations executed by the video monitoring system.FIG. 17A illustrates a scenario that may be observed in the environment100 of FIG. 1, and FIG. 17A represent a simplified video frame 360captured by video camera 210 and passed to processor system 220. In thisvideo frame, caregiver 140 can be seen to have entered the hospital roomoccupied by James Williams, our vulnerable person 110. Mr. Williams isseen lying in his bed 121. Caregiver 140 is standing. Referring to FIG.18, Mr. Williams is noted to be at risk from falling and pressureulcers, and precautions are in order for fall detection, fall prevention(forecasting a possible fall event) and pressure ulcers. A moreexpansive view of Mr. Williams' medical status can be seen in FIG. 19,which notes Mr. Williams may get out of bed on his own (an increase inrisk of falling) in the early morning.

Returning to FIG. 17A, Mr. Williams 110 is recognized by execution oftrained facial recognition algorithm 317. In an aspect of tis execution,the algorithm 317 may construct bounding box 363 in the recognized headregion of Mr. Williams 110 and apply facial recognition features 365 tothe image captured of Mr. Williams. Mr. Williams' posture (lying) isdetermined by posture recognition algorithm 322 of FIG. 3 to positivelyidentify Mr. Williams' status. Similarly, the algorithm 317 constructsbounding box 363′ and applies facial recognition features 365′ topositively identify caregiver 140. Caregiver 140 also is encompassed bybounding box 361′. Caregiver 140 will subsequently be determined to bean authorized caregiver and thus is allowed in Mr. Williams' room. Sincecaregiver 140 is authorized, his presence in Mr. Williams' room does notcreate an event. Were caregiver 140 not authorized, or were the personnot recognized, the processor system 220 could generate an unauthorizedvisitor event, which would be provided to caregivers and observersresponsible for Mr. Williams' safety, security, and well-being. SinceMr. Williams is lying in bed 121, and the precautions listed in thechart shown in FIG. 19 are fall detection and fall precaution, Mr.Williams' posture does not constitute either a fall event or theprospect of a fall event.

FIG. 17B illustrates another scenario involving Mr. Williams, who can beseen in the simplified video frame 360A alone in his room. Boundingboxes 361 and 363 show that Mr. Williams 110 is sitting in bed 121,which puts him at risk of falling were he to stand up. The processorsystem 220 would execute to generate a fall prevention alert and providethe alert to caregiver 140 (FIG. 17A), who may be able to enter the roomin time to assist Mr. Williams and so prevent a fall.

FIGS. 18-20 illustrate example information displays generated by thevideo monitoring system 200. FIG. 18 displays alerts system report 370including alerts status 372. The alerts status 372 indicates the alertsthat may be generated/expected for Mr. Williams. FIG. 19 illustratesreport 370, but with an overlay of specific status report 374 for Mr.Williams. FIG. 20 illustrates report 370 with the addition of an alertresolution banner 376. These reports may be generated by the processorsystem 220 of FIG. 2 and may be displayed to caregivers/caretakers suchas caregiver 140 of FIG. 17A.

FIGS. 21, 22, and 24-26 are flowcharts illustrating example operationsof the video monitoring system 200 upon execution of various algorithmsdisclosed herein. The descriptions of the flowcharts also refer to theenvironment 100 shown in FIG. 1, the system and components shown inFIGS. 2 and 3 and the environments illustrated in FIGS. 5-17B. Theoperations of the flowcharts illustrated in FIGS. 22 and 24-26 beginwith calibrated cameras 210, trained algorithms, and possibly knownimage objection location data, and other data, stored with processorsystem 220. FIG. 21 illustrates an example operation for training thealgorithms used in the video monitoring system 200. In FIG. 21, exampleoperation 500A begins in block 380 with training a processor/algorithmusing a convolutional neural network, or similar neural network torecognize objects in an image frame (i.e., a raw image derived from theframe); to segment the image frame so as to identify, for example,floors, walls, and other relevant structures visible to a camera 210; toclassify the detected objects as to type (e.g., person, non-person,furniture, door, etc.); identify postures of person image objects;correctly relate detected objects to segments; and to track detectedobjects, frame-to-frame, among other functions. In block 385, thetrained processor/algorithm is applied to a test frame, or frames havingknown objects and segments, to verify proper and satisfactory operation.In block 390, operation of the processor/algorithm is determined to besatisfactory or not satisfactory, according to a predefined metric. Ifthe processor/algorithm perform in a manner deemed satisfactory, theoperation 500A proceeds to operation 600, and the algorithm may be usedin real-world settings such as the environment 100 of FIG. 1. Otherwise,the operation 500A returns to block 380, and further training may beattempted.

FIG. 22 is a high level view of example operation 600, which may beemployed in the environment 100 of FIG. 1 to predict and detect untowardevents related to individuals in the environment 100 and to providealerts as appropriated based on the predictions and detections.Operation 600 begins in block 610 with a frame/image capture operation.Referring to FIGS. 1-3, the operation of block 610 may include a cameramanager (a component of processor system 220) directing camera 210 inenvironment 100 to send one or more frames to processor system 220,where the frame may be processed using the components shown in FIG. 3.Alternately, or in addition, cameral 210 may send frames to processorsystem 220 automatically, based on, for example, detected movement of aperson image object in the environment, and/or, periodically, such asevery 15 minutes, for example. Following frame/image capture, operation600 moves to operation 700, and the processor system 220 executes anumber of image object analyzes, or image objectification. The output ofoperation 700, which is shown in more detail in FIG. 25, is a final setof known image objects 530 _(i-n). In operation 800, a rule set isapplied to the final set of known image objects 530 _(i-n) in order toprovide predictions of events and detection of events. The operation 800is shown in more detail in FIG. 26. Following block 800, operation 600moves to operation 900, and in block 910, if an event is sufficient totrigger an alert, the video monitoring system 200 provides alerts, block920, as appropriate, to personnel responsible for the well-being of theindividual(s) in environment 100. The operation 600 then repeats, blocks610-920.

FIG. 23 illustrates example preliminary image object 510 in apreliminary set of known image objects 510 _(i-n) derived from a frame.Example preliminary image object 510 in the preliminary set of knownimage objects 510 _(i-n) may be a person image object or a non-personimage object. Preliminary image object 510 is not, however, related toany specific image segment. Image segmentation and relation ofpreliminary image objects 510 to image segments is disclosed herein.Preliminary image object 510 may be seen to include box 512 (i.e., abounding box). Box 512, once determined, may be fixed in relation to theobject that box 512 bounds, except that when the perceiveddimensions/orientations of the bounded image change, the box 512 maychange dimensions/orientation. For example, if a person image objectmoves from a sitting to a standing position, corresponding box 512 maychange to reflect the movement. Once box 512 is established, theprocessor system 220 may assign a confidence level to the box 512. Theconfidence level reflects the processor system's estimate as to thereliability of the bounded preliminary image object 510, including itsclassification (person, non-person, specific person, posture, etc.) andthe correctness of the boundary represented by the box 512. In anaspect, preliminary image objects 510 that cannot be bounded within apredetermined confidence level may be deleted from the preliminary setof image objects 510 _(i-n). Thus, a preliminary image object 510 is aknown image object. A preliminary image object 510 that is a personimage object may include person details 514. Person details 514 aredisclosed in more detail herein, including with respect to FIG. 24.Next, a preliminary image object 510 includes tracker 516. Tracker 516is a feature that allows the processor system 220 to follow an imageobject from frame to frame. Finally, a preliminary image object 510includes object details 518. Thus, a preliminary image object 510contains and/or describes features and characteristics of an object in aframe (the bounding box, object classification, person details,including the person's pose or posture, object tracking information, anda history of the bounding box (e.g., its location, shape, size,orientation). These features and characteristics enable the processorsystem 220 to follow image objects through sequences of frames (i.e., astime continues) and to access information about a specific image object;for example, if the image object is a person image object, the person'sname and pose. Thus, image object tracking may be employed to matchknown image objects with new, recently identified image objects in acurrent frame being analyzed. A recently identified image object thatcan be matched to an existing object may be added to the preliminary setof known image objects 510 _(i-n), or to a final set of image objects(see FIG. 25). Any known image objects that no longer are identified bythe image object detection process may be moved a “lost” status, where alast known position of an image object may be stored. A “lost” imageobject may be restored if the “lost” image object later is reidentifiedby the processor system 220; for example, if a “lost” person imageobject is identified in a later frame of a camera feed, the person imageobject, and its associated person details 514 may be restored in anearlier frame.

FIG. 24 illustrates a more detailed version of aspects of operation 600of FIG. 22. In FIG. 24, and referring to the example environment, systemand components of FIGS. 1-3, respectively, operation 600A includes amerge operation (block 750), an apply rule set operation 800, followedby an alert operation 900. As before with operation 600, to produce aset of known image objects (as shown in FIG. 24, preliminary set ofimage objects 510 _(i-n)) operation 600A begins with one or more videocameras 210 installed in room 100 transmitting video data to processorsystem 220. For example, processor system 220 receives, and, asnecessary, preprocesses the video data (e.g., decrypting if encrypted,interpolating if decimated or compressed) received from the one or morevideo cameras 210. Use of multiple cameras in a single room 100 such asshown in FIG. 1 allows for a greater field of view and improved accuracyof event detection. The use of multiple cameras in multiple rooms allowsfor the application of the system to a larger area. Followingpre-processing, the processor system 220 pulls individual video framesfrom video footage captured by the video camera(s) 210 to analyze videoframes for multiple parameters. Next, the processor system 220 executesobjection recognition algorithm 312 to first, detect discrete objects inthe room 100, second, identify discrete detected objects, and third todetermine locations of the identified objects in the room 100. Next, theprocessor system 220 executes object tracking algorithm 314 to followidentified image objects. The object tracking algorithm 314 may combineinformation from a history of known image object locations within room100 and may compute how an image object moves between video frames.Object tracking provides information about image objects in the videoframe, and how the image objects have moved (as appropriate). These datamay be time-stamped (with, for example, the time registered for thevideo frame), annotated with camera identification, and saved,referenced by “objects in scene.” Then, processor system 220 determinesif one or more image objects present in the room 100 is a person imageobject. If no identified image object in the room 100 is a person, theoperation 600A may retrieve another video frame.

If one or more of the identified objects is determined to be a personimage object, the processor system 220 executes trained facialrecognition algorithm 317 to identify the person image object. Toidentify the person image object, the algorithm 317 may uses apre-existing database of facial identities. For example, in the room(environment) 100, Mr. Williams' face may be registered in thepre-existing database of facial identities. Additionally, caregiverssuch as caregiver 140 of FIG. 17A, may have their faces registered inthe pre-existing database of facial identities. If the person imageobject's face matches one of the facial identities in the database, theperson image object is identified. If the face of the person imageobject does not match one of the identities in the facial identitiesdatabase, or the person image object is not facing the camera 210, theperson image object may be designated as an unknown person image object.Then, the person's identity, known or unknown, may be attached to theperson image objects, and stored, referenced by “persons in scene,” andthe “persons in scene” data. Following this operation, the processorsystem 220 may execute posture recognition algorithm 322 to identify andtrack the postures of a person identified by facial recognition as wellas an unidentified person in environment 100. The algorithm 322 analyzesthe position of joints and anatomical landmarks in relation to eachother to determine the posture of the person. Since the operationalready has established the location of persons (known or unknown) inenvironment 100, the algorithm 322 may apply postural analysis to theportion of the video frame (raw video image) containing the person,allowing for a reduction in overall computation. To effectuate thisreduction in computational load and to improve accuracy of results, thealgorithm 322 may use only the data inscribed by a boundary box, such asthose shown in FIGS. 17A and 17B, encompassing the person. The algorithm322 compares the person's posture in the current video frame to theirposture in previous video frames in order to compute how a person ismoving. The person's posture and movements then may be saved (referencedby “person's posture”) and provided for use in further operations of theprocessor system 220. Following image processing, the processor system220 deletes the raw video frame in order to ensure the privacy ofpersons under observation by the video monitoring system 200. Thesesteps may be repeated for subsequent video frames. Thus, the computedinformation about the identified objects, object movements, a person'sposture, a person's identity, and a person's movement are saved with aframe number or time reference and with a camera identification forcomparison with future video frames and for the generation of long-termmetric tracking.

The processor system 220 may combine the information collected (thelocations and movement of the objects, the person's identity, and theperson's posture and movement) to make decisions about the scene byapplying predetermined ruleset(s). For example, the combined data mayshow that there is one identified person, James Williams (who is avulnerable person 110) in room 100 and that in the current video frame,Mr. Williams is sitting on an edge of his bed 121 with his feet on thefloor after previously (in prior video frames) been lying down.Additional contextual information may be supplied such as, for example,time of day (e.g., 2 am) and that between 2 a.m. and 3 a.m., Mr.Williams routinely rises to use the restroom. These combined data may besubjected to one or more predetermined rulesets. A ruleset checks a setof predetermined conditions against the combined data, and if allconditions are met, or, alternately a subset of conditions is met analert requirement may be triggered, and an alert may be generated andsent to the appropriate caregivers and/or observers. The alert mayprovide a specific textual description of the visual informationcaptured by the video camera 210. The textual description allows for thepreservation of critical information obtained visually, without a humanobserver to view captured video footage. The alert may be sent inmultiple forms, including but not limited to a call, text, or pushnotification to a mobile device, an email, a pager, or a local alarm.

FIG. 25 is a flowchart illustrating example operation 700 of the videomonitoring system 200. The example operation 700 is described withrespect to room 100 (i.e., health care environment, see, e.g., FIGS. 1and 14) and video monitoring system 200 and components 300 (see FIGS. 2and 3, respectively). Operation 700 is executed to produce a final setof known image objects (set 530 i-n) that may be used, in combination,with image segments, to determine the status of a given individual,predict a behavior or threat to the individual (in the example, Mr.Williams 110), and provide alerts, as appropriate. For example,operation 700 may be executed to predict a possible fall by Mr. Williams110, and to detect a fall, should one occur. A video camera 210 ₁ placedin Mr. William's room 100 operates with a view of bed 121. The videocamera 210 ₁ captures video footage (frames) of Mr. Williams in the room100, from which image objects may be derived. The image objects mayinclude person image objects and non-person image objects. In anexample, non-person image objects are inanimate objects. In otherexamples, non-person image objects may be of other living objects (i.e.,animals). Operation 700 begins, block 705, with components of the videomonitoring system 200 managing (e.g., communicating with) cameras 210₁-210 _(n) to get a frame, which results in (provides or generates,block 710) raw image 505. In the example operation 700, camera 210 ₁ isin operation in room 100. Raw image 505 then is processed through twopaths, a segmentation path and an object detection path. In block 715,the video monitoring system 200 and components 300 process the raw image505 to generate one or more image segments. For example, raw image 505may be segmented, block 715, to generate a first segment representingfloor 125 of room 100 and one or more additional segments representingwalls of room 100 that are in view of camera 210 ₁. As disclosed herein,the image segmentation of block 715 may be executed using a mask or atrained neural network, for example. The output of block 715 issegmented room 520. In block 720, the video monitoring system 200 andcomponents 300 process the raw image 505 to detect objects therein andoptionally, to classify the detected objects. Detecting (andclassifying) image objects in the raw image 505 allows the processorsystem 220 to apply bounding boxes, such as the box 131 for bed 121; seeFIG. 14. Optionally in block 720, the processor system 220 computes aconfidence level that indicates the assigned box accurately encloses theimage object within the bounding box (e.g., the confidence levelassociated with box 512 of FIG. 23). Any confidence level below aconfigurable threshold may result in the processor system 220 removingthe detected image object from a potential set of known image objects.

In block 725, the segmented room data 520 and the detected image objectsare assigned a track feature or characteristic (e.g., tracker 516 ofFIG. 23). The tracker feature enables the processor system 220 to trackimage objects as well as segments through sequences of frames (i.e., astime continues) and to access information about a specific image object;for example, if the image object is a person image object, the person'sname and posture. Thus, image object tracking may be employed to matchknown image objects with the new, recently identified image objects. Arecently identified image object that can be matched to an existing(known) image object may be added to a set of known image objects. Theoutput of block 725 is a preliminary set of image objects 510 _(i-n).Any known image objects that no longer are identified by the imageobject detection process may be moved a “lost” status, where a lastknown position of an image object may be stored. A “lost” image objectmay be restored if the “lost” image object later is reidentified by theprocessor system 220; for example, if a “lost” person image object isidentified in a later frame of a camera feed, the person image object,and its associated person details 514 may be restored in an earlierframe.

Following block 725, operation 700 moves to block 730, and processorsystem 220 executes a number of algorithms (see, e.g., FIG. 3) toidentify all person image objects, if not already done and to segregateall image objects into known person image objects 510P_(i-n) and knownother (non-person) image objects 510NP_(i-n). The known person imageobjects 510P_(i-n) are processed through facial recognition (block 735)and posture detection (block 740). In an aspect, using the bounding boxfrom the person image objects, the raw image is cropped to only containa raw image of that person. The cropped image then is passed into facialrecognition, block 735, to identify the person. As part of block 735,once a person is identified, that information may be stored with thecorresponding person image object. Next, the same raw image is passed toposture detection, block 740. The resultant posture is also is storedwith the person image object as part of block 740. In an aspect,following blocks 735 and 740, if not already done, or to update theanalysis, the processor system 220 computes (updates) a confidence levelthat indicates the assigned box and corresponding classification andidentity of the person image object within the bounding box (e.g., theconfidence level associated with box 512 and person details 514 of FIG.23). Any confidence level below a configurable threshold may result inthe processor system 220 removing the detected image object from apotential set of known image objects. However, in an aspect, personimage objects that are classified as unknown may be retained. Followingblock 730 and 740, operation 700 moves to block 745 and all imageobjects are combined into a final set of known image objects 530 _(i-n).Once the final set of image objects 530 _(i-n) is generated, the finalset 530 _(i-n) is combined, block 750, with the room segments (segmentroom 520), and the image frame (obtained in block 710, and from whichthe final set 530 _(i-n) was derived) is deleted, block 760. Thus,system 200 has no access to the raw image the frame has been deleted inblock 760. The system 200 does, however, have access to the final set ofknown image objects 530 _(i-n), which contains all the relevant imageobjects in the raw image, as well as their locations as those locationschanged over the last x images, the image object classification, and ifthe know image object is a person image object, the person's name(possibly) and the person's pose or posture as that posture changed overthe last x images. Using these data, the processor system 220 executes aseries of checks, operation 800, to determine whether a person is/wassitting, standing, fallen, or needs/needed moving to prevent ulcers, orto detect/prevent/warn of other events related to the well-being of theperson. If multiple persons are present, and if one person is identifiedby the processor system 220 as a caretaker, the processor system 220 mayend or pause operation 700 with respect to the specific room 100 andperson 110. However, the operation 700 may restart once the processorsystem 220 detects the presence of only the person 110 in room 100.

FIG. 26 illustrates an example of operation 800. In FIG. 26, operation800 begins after generation of the data provided through operation ofblock 750 of FIG. 25, namely the segmented room data 520 and the finalknown image object set 530 _(i-n). Operation 800 includes a series ofprocess steps that in effect apply known rule sets to the data fromblock 750. Five series of process steps are illustrated; however, otherrule sets also may be applied, and the five series of process steps arefor illustration and are not limiting. The process steps all are shownwith an implicit “YES” to continue. Failure to satisfy (“YES”) thestated condition is an implicit “NO,” which terminates the currentseries of process steps and results in selection of the next sequentialseries of process steps. The operation 800 may execute until all fiveseries of process steps reach an implicit “NO” or reach a final step inthe series. Turning first to execution of process block 810, theprocessor system 220 determines if the person image object is NOT acaregiver/caretaker (that is, is the person image object a patient suchas Mr. Williams 110 or the person image object is unknown (notidentifiable)). If the person image object is not a caregiver/caretakeror is not identifiable, the process continues to block 812, and theprocessor system 220 determines if the non-caregiver is sitting.Determination of sitting may use the posture of the person image objectas well as other person image object details; this determination alsomay use an appropriate relation between the person image object and anyappropriate non-person image object such as chair 123. Note also thatthe person may be sitting in bed 121 or on the floor 125. See, e.g.,FIG. 14. If the non-caregiver is sitting, the process continues to block814, and the processor system 220 determines that an alert, patientsitting, should be provided. However, in block 810, if the person imageobject were determined to be a caregiver/caretaker, in an aspect,operation 800 may proceed to the process series beginning with block850, skipping the intervening process series. Turning second to block820 (noting again, that operation 800 may commence with block 820 ratherthan block 810), the processor system 220 determines if the person imageobject intersects the floor; that is, is some portion of the personimage object in contact with the floor. If the person image objectintersects the floor in block 820, the operation 800 moves to block 822and the processor system 220 determines if the person image object isnot a caretaker. If the person image object is not a caretaker (“YES” atblock 822), the operation 800 moves to block 824, and the processorsystem 220 determines if the posture of the person image object (knownpatient or unknown person), the operation 800 process to block 826 togenerate a patient standing warning. Similar processes are shown inprocess series beginning with blocks 830 and 840. In the process seriesbeginning with block 850, the processor system 220 determines if thereare two or more person-image objects detected in the frame. If two ormore person-image objects are detected, operation 800 moves to block852, and the processor determines if at least one of the person imageobjects is identifiable as a caretaker, such as the caretaker 140 ofFIG. 17A. If at least one caretaker is identified, the processor system220 may suspend retrieval and processing of image frames from camera 210and may, block 854, provide a notification that room 100 includemultiple persons.

The operations disclosed in the foregoing flowcharts may be used, forexample, to detect a patient (i.e., our vulnerable person 110—JamesWilliams) fall in the health care environment 100 of FIG. 1. A videocamera 210 placed in Mr. William's room operates with a view of bed 121.The video camera 210 captures video footage (frames) of Mr. Williams inthe room. The operations rely on data from derived from, orcorresponding to “objects in a scene,” “persons in a scene,” and a“person's posture.” Here, “scene,” “frame,” and “raw image” may be usedinterchangeably. Note that in FIG. 1, only one person, Mr. Williams, is“in the scene.” However, in some scenarios, additional persons (doctors,nurses, etc.) could be “in the scene.” Thus, processor system 220 mayexecute machine instructions to extract the “objects in a scene,” which,referring again to FIG. 1, are bed 121, clock 122, chairs 123 and 124,and floor 125. The processor system 220 also determines personidentities (known, unknown, patient, caregiver/caretaker, Mr. Williams)from the “persons in a scene.” Note, however, that the video monitoringsystem 200 may not be able to identify any person as Mr. Williams whenin fact Mr. Williams is present in the room. This situation could ariseif, for example, Mr. Williams' back was turned to the video camera 210such that the processor system 220 could not successfully apply facialrecognition processor 315 to Mr. Williams. As noted, in this scenario,operation 550 may simply end for the current video frame, and insubsequent video frames, Mr. Williams may present his face to the videocamera 210. However, Mr. Williams may have fallen, and may be dazed orunconscious, and subsequent frames may not show his face. Of course,this non-recognition scenario may be addressed by using multiple videocameras. Alternately, the components of processor system 220 may inferthe unrecognized, and hence unknown, person is Mr. Williams by use oflocation data for Mr. Williams from prior video frames. In particular,once a person in a scene is identified as Mr. Williams, the processorsystem may use motion algorithm 324 to confirm Mr. Williams' identity insubsequent video frames. Next, the processor system 220 extractsframe-to-frame motion from the “person's posture.” Assume that the datashows Mr. Williams horizontal and lying down. Furthermore, assumecontextual analysis show Mr. Williams lying on floor 125. The processorsystem 220 discards the current video frame and then uses Mr. William'sposture, his movements, and his relation to objects in the room todetect if a fall occurred. For example, to detect if a person hasfallen, processor system 220 may first check if the image objectsbounding box intersects the floor from the segmented image. Then theprocessor system 220 may check if that person is not a caretaker usingthe person details that were saved from the facial recognition, finally,we can pass the person's posture through a neural network which gives aBoolean value on whether the person has fallen or not. Thus, afterdiscarding a current video frame, the processor system 220 compares, thecomputed information for the current frame to computed information fromprevious video frames. The previous video frame information shows thatMr. Williams was standing and that the he transitioned from standingposition to lying down at a velocity and acceleration greater than thepredetermined threshold for falling. Thus, the operation executed byprocessor system 220 shows that Mr. Williams fell on the floor 125. Theprocessor system 220 then applies rulesets for Mr. Williams anddetermines the conditions for falling have been met and that falling isone of the rulesets designated for Mr. Williams. The example discussedabove detects the occurrence of a fall by Mr. Williams. However, theexample may extend to a multitude of scenarios, some of which are clearfrom FIG. 26. For example, while the ruleset for falls discussed abovedescribes a reactive alert to falls, a preemptive alert may be sent bymodifying the ruleset. To illustrate, if a vulnerable person isdetermined by a caregiver to be a high fall risk, an additional rulesetmay be set to generate an alert for getting out of bed unassisted. Oncethe processor system 220 determines the vulnerable person's posturetransitioned from lying down in bed to sitting up to standing up, apreemptive alert may be sent to allow a caregiver to enter the room andassist the vulnerable person before the vulnerable person suffers from afall.

Another example is the detection of a person having a generalizedtonic-clonic seizure. The processor system 220 can detect the anatomicallandmarks and joints of the person, compute the motion of the person'sjoints and anatomical landmarks, and compute the variance in motion. Ifthe variance in motion exceeds a threshold, the processor system 220 candetermine the person is having a seizure and can generate an alert tothe designated caregiver.

Yet another example is the prevention of pressure ulcers. Pressureulcers are a major concern to persons who are immobilized for extendedperiods of time. If a person maintains constant pressure on an area ofthe body for greater than two hours without a shift in weight, that areaof the body does not receive adequate blood flow. The lack of blood flowcauses death of tissues in that area of the body, which leads to apressure ulcer. Prevention of pressure ulcers centers on the timelyshifting of the immobilized person's weight to allow for the return ofblood flow to areas that were previously under pressure from theperson's body weight. The video monitoring system can be used to preventpressure ulcers by generating alerts to designated caregivers if aperson's weight has not been shifted within the desired time interval.

As noted herein, the video monitoring system 200 of FIG. 2 may employmultiple video cameras 210 to capture video frames for a specific roomor area. Multiple video cameras 210 may enhance the ability of theprocessor system 220 to detect accurately events and/or to forecastevents. When multiple video cameras 210 are used in this fashion, thevideo feeds from a camera may be synchronized to a common time referenceso that video frames captured by different video cameras 210 at the sametime may be compared, and to allow comparison of current video framedata extracted from a first video camera feed to be compared to videoframe data extracted from previous video frames of a second video camerafeed. In addition, the processor system 220 may include algorithms andinstructions to resolves any conflicts in extracted video frame datafrom different video camera feeds. For example, a first video camerafeed may provide a positive identification of vulnerable person 110while a second video camera feed might show the same vulnerable person110 as unknown due to lack of facial recognition resulting from anon-optimum camera angle.

The reliability of the video monitoring system 200 may be increased byimplementing periodic health checks of the video cameras 210 and othercomponents of the video monitoring system 200. Reliability may beincreased by making the video cameras 210 incapable of manipulation oroperation from the room or area in which they are installed, and byinstalling the cameras such that their optics cannot be covered by atowel, blanket, or similar object. Furthermore, the video cameras may beconfigured to provide a loss of image alert should the optics be coveredor in other circumstances in which the video camera 210 is not able totransmit video frames or a person attempts to tamper with the camera210. Still further, the video cameras 210 may be configured to send aloss of normal power signal to the processor system 220 upon loss ofwired power.

In some installations, the video cameras 210 may be configured to recordvideo frames periodically. In some installations, the video cameras 210may incorporate a steerable optics system. In an aspect, the steerableoptics system may be steered (left/right, up/down) by executing asteering program. In another aspect, the steerable optics system may, inaddition, be manually steerable, but will return to programmed steering,or will return to a predetermined angle at a set time after manualsteering operations. In some installations, the video camera 210 mayincorporate zoom optics. In some installations, the video camera 210 maycompress the video files using, for example, compression processes, inorder to reduce transmission bandwidth.

In some installations, the processor system 220 may generate, provide,and display an alert upon loss of video feed from all cameras 210installed in a room or area.

The video monitoring system 200 may be applied in a personal homesetting as well as the environment 100 shown in FIG. 1. In this homeenvironment, vulnerable persons normally require a caregiver to monitorthem and assist them with activities of daily living. Caregiving can bea stressful, time consuming, and tiresome job. The video monitoringsystem 200 provides benefits to both the vulnerable person as well asthe caregiver. For example, it is not feasible for a caregiver tophysically watch the vulnerable person at all times. At some point, acaregiver will have to leave the vulnerable person to perform otherduties or tasks, such as use the restroom, cook, or sleep. When thecaregiver must leave the side of the vulnerable person, there is a riskthat the vulnerable person will do something that risks of injury or anunwanted outcome. For example, a vulnerable person with dementia mayfall or attempt to use a dangerous household object or appliance such asa knife, stove, or fireplace. Use of the such devices or appliances maypose risk to a person with dementia because of self-injury, or damage ordestruction of the home. Furthermore, persons with dementia are at riskby wandering away from the caregiver. In addition to the risks to thevulnerable person, there is an associated high level of stress for thecaregiver. The video monitoring system 200 addresses these problems byproviding continuous, uninterrupted monitoring of the vulnerable person,along with an instantaneous alert to the caregiver in the event of anadverse or undesired such as a fall, use of certain objects orappliances, or exiting a room or the house. The video monitoring system200 thus provides safety to the vulnerable person by ensuring constantmonitoring, and provides peace of mind to the caregiver when leaving theside of the vulnerable person, thereby reducing stress for thecaregiver.

In addition to the technological advances of current video monitoringsystems disclosed herein, the video monitoring system 200 providesadditional advances of current systems that may be employed the workflowof caregivers. For example, using the facial recognition processor 315for both caregivers and patients, the system 200 may integrate withelectronic medical record (EMR) systems used by healthcare facilities toautomatically sign in a caregiver into the EMR and locate a patientchart using staff credentials and patient profile pictures in the EMR.This will improve efficiency for caregivers by forgoing manual log-ins.Furthermore, the system 200 allows for an automated and objective way oftracking patient-caregiver encounters. For example, at a certain time,nurse X interacted with patient Y. At a later time, nurse Z interactedwith patient Y. The system 200 provides accurate and objectiveinformation about patient-caregiver encounters and ensuresaccountability.

The video monitoring system 200 also provides for longitudinal,automated, objective metric tracking that is significantly improved overcurrent systems and methods. The objective metric tracking may beapplied in various situations, including but not limited torehabilitation, developmental milestones, and chronic degenerativediseases. Metrics that may be tracked include, but are not limited to,steps taken, distance walked, sleep activity, and changes in gait(stride length, speed, posture, symmetry). Objective data that isconstantly collected over an extended period of time may be used toidentify trends, and thus identify changes in a person's condition, suchas progress in rehabilitation, or deterioration with a degenerativedisease.

Certain of the devices shown in FIGS. 2-4 include a computing system.The computing system includes a processor (CPU) and a system bus thatcouples various system components including a system memory such as readonly memory (ROM) and random-access memory (RAM), to the processor.Other system memory may be available for use as well. The computingsystem may include more than one processor or a group or cluster ofcomputing system networked together to provide greater processingcapability. The system bus may be any of several types of bus structuresincluding a memory bus or memory controller, a peripheral bus, and alocal bus using any of a variety of bus architectures. A basicinput/output (BIOS) stored in the ROM or the like, may provide basicroutines that help to transfer information between elements within thecomputing system, such as during start-up. The computing system furtherincludes data stores, which maintain a database according to knowndatabase management systems. The data stores may be embodied in manyforms, such as a hard disk drive, a magnetic disk drive, an optical diskdrive, tape drive, or another type of computer readable media which canstore data that are accessible by the processor, such as magneticcassettes, flash memory cards, digital versatile disks, cartridges,random access memories (RAM) and, read only memory (ROM). The datastores may be connected to the system bus by a drive interface. The datastores provide nonvolatile storage of computer readable instructions,data structures, program modules and other data for the computingsystem.

To enable human (and in some instances, machine) user interaction, thecomputing system may include an input device, such as a microphone forspeech and audio, a touch sensitive screen for gesture or graphicalinput, keyboard, mouse, motion input, and so forth. An output device caninclude one or more of a number of output mechanisms. In some instances,multimodal systems enable a user to provide multiple types of input tocommunicate with the computing system. A communications interfacegenerally enables the computing device system to communicate with one ormore other computing devices using various communication and networkprotocols.

The preceding disclosure refers to flowcharts and accompanyingdescription to illustrate the examples represented in FIGS. 21, 22, and24-26. The disclosed devices, components, and systems contemplate usingor implementing any suitable technique for performing the stepsillustrated. Thus, FIGS. 21, 22, and 24-25 are for illustration purposesonly and the described or similar steps may be performed at anyappropriate time, including concurrently, individually, or incombination. In addition, many of the steps in the flow chart may takeplace simultaneously and/or in different orders than as shown anddescribed. Moreover, the disclosed systems may use processes and methodswith additional, fewer, and/or different steps.

Examples disclosed herein can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including theherein disclosed structures and their equivalents. Some examples may beimplemented as one or more computer programs, i.e., one or more modulesof computer program instructions, encoded on computer storage medium forexecution by one or more processors. A computer storage medium can be,or can be included in, a computer-readable storage device, acomputer-readable storage substrate, or a random or serial accessmemory. The computer storage medium can also be, or can be included in,one or more separate physical components or media such as multiple CDs,disks, or other storage devices. The computer readable storage mediumdoes not include a transitory signal.

The herein disclosed methods can be implemented as operations performedby a processor on data stored on one or more computer-readable storagedevices or received from other sources.

A computer program (also known as a program, module, engine, software,software application, script, or code) can be written in any form ofprogramming language, including compiled or interpreted languages,declarative or procedural languages, and it can be deployed in any form,including as a stand-alone program or as a module, component,subroutine, object, or other unit suitable for use in a computingenvironment. A computer program may, but need not, correspond to a filein a file system. A program can be stored in a portion of a file thatholds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub-programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

We claim:
 1. A system for monitoring safety, security, and well-being ofa person, comprising: a non-transitory, computer-readable storage mediumhaving encoded thereon, machine instructions executable by a processorto monitor the safety, security, and well being of the person, whereinthe processor: receives raw image data of video frames captured by oneor more of a plurality of video cameras; detects and identifies imageobjects in a current received video frame; classifies an identifiedimage object as a vulnerable person, comprising the processor applying afacial recognition algorithm to the identified image object; determinesa posture of the vulnerable person, comprising identifying a pluralityof body parts of the vulnerable person, and respective orientations ofthe body parts; provides vulnerable person privacy, comprisingautomatically and immediately following detecting and classifying theimage object as the vulnerable person and determining the posture of thevulnerable person, deletes the current received video frame; and basedon the determined posture, determines that the vulnerable person mayexperience a first defined event.
 2. The system of claim 1, wherein todetermine a posture of the vulnerable person, the processor compares therespective orientations of the body parts relative to one or more of avertical plane and a horizontal plane.
 3. The system of claim 2, whereinto determine a posture of the vulnerable person, the processor:generates a segmented room comprising one or more room segments, a roomsegment corresponding to a surface of the segmented room; and determinesa relationship between the vulnerable person and one or more of the roomsegments.
 4. The system of claim 3, wherein a segment comprises a floorsegment of the segmented room, and wherein the image object classifiedas the vulnerable person, contacts at least a portion of the floorsegment.
 5. The system of claim 2, wherein the processor determines anamount of change in posture of the vulnerable person between the currentreceived video frame and one or more prior received video frames; andbased on the determined amount of change in posture of the vulnerableperson, determines the vulnerable person has experienced a seconddefined event, wherein the second defined event is a fall.
 6. The systemof claim 2, wherein the processor determines an amount of change inposture of the vulnerable person between the current received videoframe and one or more prior received video frames; and based on thedetermined amount of change in posture of the vulnerable person,determines the vulnerable person may experience a third defined event,wherein the third defined event is a pressure ulcer.
 7. The system ofclaim 1, wherein to classify the identified object as the vulnerableperson, the processor: generates a bounding box encompassing theidentified image object; and applies the facial recognition algorithmonly to the identified image object encompassed in the bounding box. 8.The system of claim 1, wherein the first defined event is a fall, andwherein the processor generates a potential fall alert and provides thepotential fall alert to a caretaker attending to the vulnerable person.9. The system of claim 1, wherein the processor: generates a confidencelevel for accuracy of the classifying; determines the generatedconfidence level is less than a desired confidence level; and deletesthe image object.
 10. A method for monitoring safety, security, andwell-being of a person, comprising: a processor receiving raw image dataof video frames captured by one or more of a plurality of video cameras;detecting and identifying image objects in a current received videoframe; classifying an identified image object as a vulnerable person,comprising the processor applying a facial recognition algorithm to theidentified image object; determining a posture of the vulnerable person,comprising identifying a plurality of body parts of the vulnerableperson, and respective orientations of the body parts; providingvulnerable person privacy, comprising automatically and immediatelyfollowing detecting and classifying the image object as the vulnerableperson and determining the posture of the vulnerable person, deletingthe current received video frame; and based on the determined posture,determining that the vulnerable person may experience a first definedevent.
 11. The method of claim 10, comprising determining a posture ofthe vulnerable person by comparing the respective orientations of thebody parts relative to one or more of a vertical plane and a horizontalplane.
 12. The method of claim 10, comprising determining a posture ofthe vulnerable person, by: generating a segmented room comprising one ormore room segments, a room segment corresponding to a surface of thesegmented room; and determining a relationship between the vulnerableperson and one or more of the room segments.
 13. The method of claim 12,wherein a segment comprises a floor segment of the segmented room, andwherein the image object classified as the vulnerable person, contactsat least a portion of the floor segment.
 14. The method of claim 10,comprising: determining an amount of change in posture of the vulnerableperson between the current received video frame and one or more priorreceived video frames; and based on the determined amount of change inposture of the vulnerable person, determining the vulnerable person hasexperienced a second defined event, wherein the second defined event isa fall.
 15. The method of claim 10, comprising: determining an amount ofchange in posture of the vulnerable person between the current receivedvideo frame and one or more prior received video frames; and based onthe determined amount of change in posture of the vulnerable person,determining the vulnerable person may experience a third defined event,wherein the third defined event is a pressure ulcer.
 16. The method ofclaim 10, comprising classifying the image object as the vulnerableperson by: generating a bounding box encompassing the identified imageobject; and applying the facial recognition algorithm only to theidentified image object encompassed in the bounding box.
 17. The methodof claim 16, comprising: generating a confidence level for an accuracyof the classifying; determining the generated confidence level is lessthan a desired confidence level; and deleting the image object.
 18. Themethod of claim 10, wherein the first defined event is a fall, themethod comprising: generating a potential fall alert; and providing thepotential fall alert to a caretaker attending to the vulnerable person.19. A system for predicting an occurrence of events harmful to amonitored, vulnerable person, comprising: a plurality of video camerasarranged to monitor the vulnerable person; and a processor systemconfigured to receive video frames from a video stream provided by theplurality of video cameras, the processor system comprising one or moreprocessors and a non-transitory, computer-readable storage medium havingencoded thereon machine instructions executed by the one or moreprocessors, wherein a processor: detects and identifies image objects ina current received video frame, classifies an identified object as thevulnerable person, comprising: applying a bounding box to the identifiedobject; and applying a facial recognition algorithm only to theidentified object encompassed by the bounding box, the facialrecognition algorithm comparing facial data to a database of facialimages including the vulnerable person, determines a posture of thevulnerable person, comprising identifying joints, limbs, and body parts,and their respective orientations to each other and to one or more of ahorizontal plane and a vertical plane, providing vulnerable personprivacy comprising automatically and immediately following thedetecting, classifying, and determining, deleting the current receivedvideo frame, and based on the determined posture determines that thevulnerable person may experience a defined event.
 20. The system ofclaim 19, wherein the processor: determines an amount of change inposture of the vulnerable person between the current received videoframe and one or more prior received video frames; and based on thedetermined amount of change in posture of the vulnerable person,determining the vulnerable person has experienced the defined event.